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ABSTRACT 


A connected hypercube with faulty links and/or nodes is called an injured hypercube To 
enable any non-faulty node to communicate with any other non-faulty node in an injured hyp 
cube, the ^formation on component failures has to be made available to non-faulty nodes so as to 
route messages around the faulty components. 

W© propose first a distributed adaptive fault-tolerant routing scheme for an injured hyper- 
cube in which each node is required to know only the condition of its own [ ,nks - 
plicity, this scheme is shown to be capable of routing messages successfully nan injured hyper- 
cube as long as the number of faulty components is less than n. Moreover it is proved that t 
scheme routes messages via shortest paths with a rather high probability and the expected length 
of a rosuTtSng path is very close to that of a shortest path. Since the assumption that the number 
of faulty components is less than n in an n-dimensional hypercube might limit the l.„ 

the above scheme, we also introduce a routing scheme based on depth-first search which works 
the presence of an arbitrary number of faulty components. 

Due to the insufficient information on faulty components, however, the paths chosen by thc 
above scheme may not always be the shortest. To guarantee all messages to be routed via shor- 
test paths we propose to equip every node with more information than that on its own links. The 
effects of this additional information on routing efficiency are analyzed, and the additional infor- 
mation to be kept at each node for the shortest path routing is determined. Several examples and 
remarks are also given to illustrate our results. 


Index Terms : Injured and regular hypercubes, distributed adaptive fault-tolerant routing, depth- 
first search, looping effects, network delay tables, failure information. 
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1. INTRODUCTION 

In recent years, advances of VLSI and computer networking technologies have made it 
attractive to build multicomputer systems for numerous applications. Hypercube multi computers, 
among others, have been drawing considerable attention due mainly to their structural regularity 
for easy construction and high potential for the parallel execution of various algorithms ( 1 , 2, 3], 
Numerous research projects related to hypercube architectures, operating systems, and program- 
ming languages have been undertaken [4, 5, 6, 7, 8, 9], and several research and commercial 
hypercube multicomputers have been built [10, 11]. 

Efficient routing of messages is a key to the performance of a multicomputer system. Espe- 
cially. the increasing use of multicomputer systems for reliability-critical applications has made it 
essential to design fault-tolerant routing strategies for such systems. By fault-tolerant routing, we 
mean the successful routing of messages between any pair of non-faulty nodes in the presence ol 
faulty components Ginks and/or nodes). Several important results on the fault-tolerant routing in 
various networks have been reported [12,13,14], A few architectures are also proposed and 
shown to possess fault-tolerant routing capabilities [15. 16, 17. 18]. When hypercubc multicom- 
puter are to be used for reliable applications, they must be made to be able to route messages 
even in the presence of faulty links and nodes. 

A connected hypercube with faulty components is called an injured hypercube, whereas a 
hypercube without faulty components is called a regular hypercube. It is well-known that rout- 
ing in a regular hypercube can be handled by a systematic procedure [9], Some results on 
improving the routing efficiency in regular hypercubes are reported in [19], Routing in an mcom- 
plete hypercube is also shown to be straightforward [20], In a regular hypercubc, each intermedi- 
ate node can determine the next hop of a message by examining the message’s destination 
address and choosing, from all its neighboring nodes, the one which is closest to the destination, 
dearly, this can be accomplished by aligning the address of the source node with that of the dcs- 
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tination node from right to left bit-by-bit. However, this scheme becomes invalid in an injured 
hypercube, since the message may be routed to a faulty component. In order to enable non-faulty 
nodes in an injured hypercube to communicate with one another, enough network information has 
to be incorporated into either the message to be routed or each node in the network so as to route 
messages around the faulty components. Using additional hardware called a hyperswitch, a best- 
first search algorithm for routing messages in a hypercube is developed in [12]. Several adaptive 
packet routing algorithms are also presented in [21], which are based on the routing strategics 
used for wide-area computer networks such as ARPANET. In addition, various algorithms arc 
proposed in [14,22] to broadcast the information about faulty components to all the other nodes 
in a hypercube so that messages can be routed around the faulty components. Clearly, if each 
node is equipped with the information on all faulty components, then it can always determine a 
fault-free path for every message to its destination. However, it is usually too costly (in space 
and time) to equip ever? node with the information on all faulty components, especially when the 
network is large. Hence, it is important to develop routing schemes which require each node to 
keep only the failure information essential for making correct routing decisions. 

For the reasons above, we shaU develop first a routing scheme which requires each node to 
know only the condition of its own links. As will be seen later, this scheme attempts to route 
every message to its destination via an optimal path which is defined as a path of length equal to 
the Hamming distance 1 between the message’s source and destination nodes. When the 
insufficient knowledge on faulty components causes a message to be sent to an intermediate node 
from which there is no optimal path to the destination node, an alternative path will be chosen in 
such a way that the connectivity of a hypercube is fully exploited, and those faulty components 
encountered before will not be encountered again in future. This scheme is proven to be capable 
of routing messages between any pair of non-faulty nodes as long as the total number of faulty 
components is less than n in an n-dimensional hypercube, Q n . More importantly, this scheme, 

'To be defined formally in the next section. 
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despite its simplicity, is shown to be very powerful in that the probability of routing messages via 
shortest paths is very high and the expected length of a resulting path is very close to the optimal 
one. To route a message in an injured hypercube, the infomtation on component failures is incor- 
porated into the message as the message travels toward the destination. This fact distinguishes 
our approach from others, such as those in [12,21], 

The assumption that the number of faulty components is less than n in an injured Q n could 
limit the usefulness of the above algorithm. To remove this limitation, we introduce a second 
routing scheme based on depth-first search which requites more provisions but works in the pres- 
ence of an arbitrary number of faulty components. Since the performance analysis we developed 
for the first routing scheme can be extended for the routing scheme based on depth-first search, 
the performance analysis of the latter is not included in this paper. 

Due to the absence of information on all faulty components in the network, the paths chosen 
by the above two schemes may not be shortest. The efficiency of routing (measured in terms of 
the length of a path chosen) can be improved if every non-faulty node is equipped with moic 
information than that on its own links, since in such a case the faulty components on the way of 
every message to its destination can be foreseen, and thus, bypassed. It can be observed that the 
abundant connections in a hypercube usually make routing decisions in a node unaffected by the 
failure of a component which is located far away from the node. Based on this observation, we 
shall develop a third routing scheme for which each node is required to keep only the infomiation 
essential for the shortest path routing of messages. The information in each node required for the 
shortest path routing will be determined in light of some properties of the hypcrcubc. Since the 
third method is again based on the assumption that the total number of faulty components is less 
than n in an injured Q n , we introduce a fourth routing scheme which uses network delay tables 
[23] and worics in the presence of an arbitrary number of faulty components. However, this 
scheme requires each node to maintain and update a delay table, which could be costly for large 

hypercubes. 
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The paper is organized as follows. Necessary notation and definitions are given in Section 
2. We shall present in Section 3 an adaptive fault-tolerant routing scheme which requires each 
node to know only the condition of its own links. An alternative routing scheme based on depth- 
first search is also introduced and discussed there. Section 4 presents two fault-tolerant routing 
schemes which require each node to include more information than that on its own links, one 
with propagation of failure information, and the other with network delay tables. Illustrative 
examples and some remarks are also given. The paper concludes with Section 5. 


2. PRELIMINARIES 

An n-dimensional hypercube (or n-cube or Q n ) is formally defined as follows. 

Definition 1: An n-cube, Q n , is defined recursively as follows. 

(i) Q 0 is a trivial graph with one node, and 

(ii) Q n = K 2 xQ n _„ 

where K 2 is the complete graph with two nodes, Q 0 is a trivial graph with one node and x is the 
product operation of two graphs [24]. 

It follows from Definition 1 that a Q n contains 2 n nodes and n2 n_1 links since the degree ol 
each node in a Q n is n. Let £ be the ternary symbol set {0, 1, *}. where * is a don’t care symbol. 
Every subcube in a Q n can then be uniquely represented by a string of symbols in £ . Such a 
string of ternary symbols is called the address of the corresponding subcube. For example, the 
address of the subcube Q 2 formed by nodes 0010, 0011, 0110 and 0111 in a Q 4 is 0*1*. Fig. 1 
shows a Q 2 with address 0*1* in a Q 4 . The rightmost coordinate of the address of a subcubc will 
be referred to as dimension / , and the second rightmost coordinate as dimension 2, and so on. For 
each hypercube node, the communication link in dimension i is called the i-th link of this node. 
For notational simplicity, each link is represented by a binary string with a symbol in the 
corresponding dimension. For example, the link between nodes 0000 and 0010 is represented by 
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00-0. Let r + and r~ denote the two end nodes of a link r, where r + (r ) represents the node whose 
address is obtained by changing the symbol in the link’s address to 0 (1). 


Definition 2: The Hamming distance between two hypercube nodes with addresses 
u = u„u n _i • • • Uj and w = w n w n _ ) • • • W] in a Qn is defined as 


H(u, w) = Xh(u„ Wi), where h(u,,w,) =\ 


i=l 


1, if Ui * Wi, 

0, ifu^w^ 


For the nature of distributed routing strategies to be presented, it is necessary to introduce 
the exclusive operation between two binary strings, and the concept of relative address between 
two hypercube nodes. 

Definition 3: The exclusive operation of two binary strings q = q n q n -i ' ' Qi an ^ 
m = m n m n _i • • • m t , denoted by q©m = r n r n _] • • • rj, is defined as q = 0 if q; = m, and q = 1 tf 
qj = mj for 1 < i < n. 

Obviously, the exclusive operation is commutative, i.e., q©m = m©q. We use©,-] to 
denote k sequential exclusive operations. The relative address of a node u with respect to another 
node w, denoted by u /w , can then be determined by u /w = u©w. The relative address of a subcubc 
with respect to a node u can be determined by the relative addresses of all the nodes it contains. 
Let e k = e^ ■ • • e] where e k = 1 and ej = 0 V j * k. For example, 1001 © e 2 = 1011, 

001 l/iooi = 1010, and 0*l* /100 i = l* 1 *- 

Definition 4: The spanning subcube of two nodes u = u„u n _] • ■ • u, and w = w n w n _, • • • w, 
in a Q n , denoted by SQ(u,w) = s„Vi ■ • • s lt is defined as % = u, if u, = w it and s ; = * if u, * w ; Tor 
1 < i < n. 

For example, when u = 0010 and w = 01 11, we get H(u,w) = 2 and SQ(u.w) = 0*1*. It is 
easy to see that SQ(u,w) is the smallest subcube that contains both u and w, and H(u,w) is the 
dimension of SQ(u,w). 
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A path in a hypercube is represented by a sequence of nodes in which every two consecu- 
tive nodes are physically adjacent to each other in the hypercube. The number of links on a path 
is called the length of the path. An optimal path is a path whose length is identical to the Ham- 
ming distance between the source and destination nodes. A shortest path is a path of minimal 
length among all fault-free paths from the source to the destination. Clearly, an optimal path is a 
shortest path, but a shortest path is not always an optimal path in an injured hypercube. Also, a 
link of node u is said to be toward any node w if this link is in an optimal path between u and w. 
Note that due to the special structure of a hypercube, once the source node of a path is given, the 
path can be described by a coordinate sequence that represents the order of the dimensions in 
which every two consecutive nodes in a path differ [25], As shown in Fig. 2, [0001, 001 1 , 0010, 
1010] is an optimal path from the source node 0001 to the destination node 1010, and can also be 
represented by a coordinate sequence [2, 1, 4], In addition, we shall assume that the source and 
destination nodes are non-faulty. 

3. ROUTING WITH INFORMATION ON LOCAL LINK FAILURES 

In this section, we first develop and analyze an adaptive routing algorithm, called Algorithm 
A lt which requires every node to know only the condition of its own links. This algorithm will 
be shown to successfully route messages between any pair of non-faulty nodes as long as the 
number of faulty components is less than n in a Q n . As mentioned earlier, the assumption of the 
total number of faulty components to be less than n in a Q n may limit the usefulness of A,. Thus, 
we shall intorduce a second routing scheme based on depth-first search which works in the pres- 
ence of an arbitrary number of faulty elements. However, due to the insufficient amount of infor- 
mation on faulty components, the paths chosen by these two algorithm may not always be the 
shortest. 
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3.1. Description of Algorithm A] 

Before describing Algorithm A,, it is necessary to introduce the following lemma which 
determines relative addresses of those nodes traversed by a given path. 

Lemma 1: Let [c lt c 2 , • • • , c k ] be the coordinate sequence of a given path in a Q n starting 
from node u. and W/u = w n w n _, • • • w, denote the relative address of node w with respect to u, 
where k = H(u,w). Then, the path specified by [c lt C 2 , • • • , c k ] ends at w if and only if 

< ^ie Ci = 

Proof: Traversal of a message along the i-th dimension is the same as inverting the bit in 
the i-th coordinate of the relative address of its destination. Therefore, traveling along a certain 
dimension an even number of times has the same effect as not traveling along that dimension at 
all, and thus, this lemma follows. Q.E.D. 

For example, a path with the coordinate sequence [3, 4, 2] from 0110 will traverse nodes 
0010, 1010 and then 1000. The following theorem, which was previously introduced in [26], is 
useful for our discussion that follows. 

Theorem 1 [26]: Let u and w be two arbitrary nodes in a Q„ such that H(u,w) = k. Then, 
there are exactly n disjoint paths of length less than or equal to k+2 from u to w. These paths arc 
composed of k disjoint paths of length k, and (n - k) disjoint paths of length k+2. 

Fig. 3 gives an example for 4 disjoint paths in a Q 4 when H(u,w) = H(0000, 01 1 1) = 3. 
Theorem 1 leads to the following corollary. 

Corollary 1.1: Let f be the number of faulty links and g be the number of faulty nodes in 
an injured Q n such that f + g < n. Then, there always exists at least one path of length less than or 
equal to k+2 between any two non-faulty nodes u and w, where H(u,w) = k. 
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We can now describe Algorithm A! as follows. To indicate the destination of a message, 
the coordinate sequence of a path is sent along with the message. Additionally, each message is 
accompanied with an n-bit vector tag = d n d n _] • • • d] which keeps track of spare dimensions 
that are used to bypass faulty components. All bits in the tag are reset to zero when the source 
node begins routing of a message. Therefore, such a message can be represented as (k, 
[C|, C 2 , • • • , c k ], message, tag), where k is the length of the remaining portion of the path and is 
updated as the message travels towards the destination. A message reaches its destination when k 
becomes zero. 

When a node receives a message, it will check the value of k to see if the node is the desti- 
nation of the message. If not, the node will try to send the message along one of those dimen- 
sions specified in the remaining coordinate sequence. (Note that the coordinate sequence will 
also be updated as the message travels through the hypercube.) Each node will attempt to route 
messages via shortest paths first. However, if all the links in those dimensions leading to shortest 
paths are faulty, the node will use a spare dimension to route the message via an alternative path. 
(Recall that the spare dimensions are kept track of by a tag.) More formally, this routing scheme 
can be described in algorithmic form as follows. 

Algorithm Fault-tolerant routing algorithm to be used by each node only with the informa- 
tion on its own links. 

/* For each node receiving (k, [Cj, c 2 , • • • , c k ], message, tag) */ 
if k=0 then {the destination is reached! } 

else begin 

/* Try to send the message along a dimension in the remaining coordinate sequence. */ 
for j := 1, k do 

if (the Cj-th link is not faulty) then - — (t) 

/* (f) is a conditional statement which will be modified later in Section 4.1. */ 
begin 

send (k-1, [c lt • • • c^.c^,, • • • , c k ], message, tag) along the c r th link; 
stop; /* terminate Algorithm Aj */ 
end_begin 
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end_do 

/* if the algorithm is not terminated yet, all dimensions in the coordinate sequence arc 
blocked because of faulty components and a spare dimension needs to be used. / 

for j := 1, k do /* record all blocked dimensions in tag. */ 

<V =1 

end_do; 

h := min (i : dj = 0, 1 < i ^ n) ; /* choose a spare dimension */ 
d h := 1; /* update the tag */ 

send (k+1. [Ci,C 2 •••.<*, h], message, tag) along the h-th link; 
stop; /* terminate Algorithm A, */ 
end_begin 

Consider the Q 4 in Fig. 4, where links 0-01, 1-01 and 100- are faulty. Suppose a message, 
fm. is routed from u = 0110 to w = 1001. The original message in u = 0110 is (4, [1,2, 3, 4], fm. 
0000). Following the execution of A„ node 01 10 sends (3, [2.3,4], fm, 0000) to node 0111 which 
then sends (2, [3,4], fm, 0000) to node 0101. Since the 3-rd dimensional link of 0101 is faulty, 
node 0101 wiU route (1, [3], fm, 0000) to 1101. However, since the 3-rd dimensional link ol 
1101 is faulty, node 1101 will use the 1st dimension (tag = 0100 then), and send (2, [3,1], fm, 
0101) to 1100, which wiU, in turn, send (1, [1], fin. 0101) to 1000. Again, the first link of node 
1000 is faulty. The 2-nd dimension (tag= 0101 then) will be used and (2, [1,2], fm, 0111) is 
routed to 1010. After this, the message will reach the destination 1001 via 1011. The length of 

the resulting path is 8. 

3.2. Performance Analysis of Algorithm 

The following theorem proves that Algorithm A ( can route messages between any two 
non-faulty nodes as long as the number of faulty components is less than n. 

Theorem 2: Algorithm A, can always route messages between any two non-faulty nodes 
successfully as long as the number of faulty components is less than n, i.e., f + g < n, where f and 
g are the numbers of faulty links and faulty nodes in a Q n , respectively. 
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Proof: Note that each node will try to use a spare dimension only when faulty components 
are encountered in all the dimensions specified by the coordinate sequence. Those faulty com- 
ponents which block the optimal paths from an intermediate node to the destination node and 
force the first use of a spare dimension are called type- A blocking components. On the other 
hand, a faulty component is said to be type-B if it is encountered first after using a new spare 
dimension. For the example routing in Fig. 4, 1-01 is a type-A blocking component and 100- is a 
type-B blocking component, whereas the faulty link 0-01 is neither type-A nor type-B. For the 
example in Fig. 5 where u = 0000 and w = 1 1 1 1, 0-1 1 and -01 1 are type-A blocking components 
while 111- is a type-B blocking component. Notice that both the types of blocking components 
can be either faulty nodes or faulty links. Thus, it is easy to see that the number of both type-A 
and type-B blocking components in the route determined by A t usually increases as the message 
is routed towards its destination. 

Let bh be a type-B blocking component which is encountered first after using a new spare 
dimension h. We claim that the blocking component b^ does not belong to the set of those block- 
ing components that had already been encountered before. This claim is proved by considering 
two possible cases of (i) bh is a link of the destination node, and (ii) b h is not a link of the des- 
tination node. In the case of (i), bh is the h -th link of the destination node. Since d h of tag was 0 
before the spare dimension h is used, this faulty link had definitely not been encountered before. 
In the case of (ii), since bh is the blocking component encountered first after using the spare 
dimension h, bh and the set of previous blocking components must be located in the two different 
Q^i’s separated by the dimension h. The claim is thus proved. Since a certain faulty component 
will not be encountered more than once as long as the number of faulty components is less than n, 
this theorem thus follows. Q.E.D. 

The corollary below follows from Theorem 2 and the fact that the number of hops is 
increased by two whenever a spare dimension is used. 
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Corollary 2.1: Suppose k spare dimensions are used for routing a message from node u to 
node w by A,. Then, the length of the resulting path is H(u,w) + 2k. 

It can be easily verified that the worst case of Algorithm Aj needs H(u,w) + 2(n-l) steps to 
send a message from u to w. To facilitate our presentation, the first node which is forced to use a 
spare dimension is called an obstructed node. For example, the obstructed nodes in the examples 
of Figs. 4 and 5 are 1 101 and 001 1, respectively. Then, we have the following lemma. 

Lemma 2: Suppose there are f faulty links in a Q„, and a message is routed by A, from 
node u to node w, where H(u,w) = k. Let m A be the Hamming distance between the obstructed 
node and the destination node. Then, P(m A =j) < C f l :"VC f L if 1 < j < k, and P(m A =j) = 0 if j > k, 
where Cy represents the combinations of choosing y out of x possibilities, and L = n2 is tire 

number of links in a Q n . 

Proof: P(m A > k) = 0, since the inequality m A > k represents an impossible case in which a 
message is not directed towards its destination before encountering the obstructed node. Con- 
sider the case of 1 < j < k and assume there are f faulty links in an injured Q n . Since these faults 
may occur at any f links in the Q n , there are Q L different configurations (of faulty links) where 
L = r^" -1 . Without loss of generality, we can let u = 0" and w = 0" k l k . The problem of obtain- 
ing P(m A =j) is then reduced to that of counting the number of configurations which lead to the 
case of m A = j. We claim that the number of such configurations is less than or equal to 

When m A =j, the obstructed node must be within the subcube 0 n k * k , and all its j links 
towards w must be faulty (i.e., j type-A blocking components). Although there are many possible 
locations of the obstructed node, according to the systematic procedure of Aj, the location of the 
obstructed node is determined by those non-type-A faulty links which are not within 0 * . Sup- 

pose x = 0 n " k+ jl k "j is the obstructed node, then the j links of node x within SQ(x,w) are faulty and 
there are C different distributions of these non-type-A faulty links. When these non-typc-A 
faulty links cause node y, instead of x, to be the obstructed node, we switch the links (including 
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faulty links) in SQ(y,w) to those in SQ(x.w), and obtain a configuration which leads to the case 
when the obstructed node is y and m A = j. Notice that some of the Cjl^ different distributions of 
non-type-A faulty links may lead to m A > j, meaning that the number of configurations leading to 
m A = j is less than or equal to C/iyf This lemma thus follows. Q.E.D. 


From Lemma 2, we can obtain the following theorem, showing that Algorithm A, can route 
a message to the destination via an optimal path with a very high probability. 

Theorem 3: Suppose there are f faulty links in a Q n . Algorithm A, will route a message 
from a node u to to another node w via an optimal path between u and w with a probability 

k ch~* 

greater than 1 - £ — where L= vQP~ x , and H(u,w) = k. 
h Q 

k 

Proof: From Lemma 2, the probability that A, has to use spare dimensions is £P(m A = j) < 


2- r i 

i=i w 


Thus, the probability that A] will not use any spare dimension at all is 1 XP( m A - j) - 
L ‘ j=l 


k C]W 

j=l w 


Q.E.D. 


When there are n-1 faulty links in a Q n , the lower bound of the probability that A! will 
result in the optimal path routing can be derived as follows. 

Corollary 3.1: Suppose there are n-1 faulty links in a Q„. Algorithm A t will route a mes- 
sage from a node u to another node w via an optimal path between u and w with a probability 

greater than 1 — , where H(u,w) = k and r x = . • 

(1 — r ( ) nz 


Proof: From Theorem 3, we have 


i. cf-ti 
h ch 


n-1 (n-l)(n-2) 

L L(L-1) 


(n-l)(n-2) • • • (n-k) wherc L = n2 
L(L-1) • • • (L-k+1) 
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Since rj = 


n-1 

L 


> 


n-2 

> 

L-l 


n-k 

L-k+1 ’ 


we get 




< r i + T i + 


H 


C „ L -1 


> 1 - 


d-ri) 


This corollary thus follows. Q.E.D. 


From Theorem 3, we derive Table 1 which shows the lower bound of the probability of A, 
routing messages via optimal paths (we shall henceforth call this optimal path routing ) between 
two nodes of Hamming distance n-1 apart in a Q n with f faulty links. It can be seen from 
Theorem 3 that A! will route a message to its destination via an optimal path with a rather high 
probability in the presence of faulty links. Notice that the expression in Theorem 3 can also be 
applied to the case that the number of faulty links is greater than n in a Q n . This is the very reason 
that we included such cases as n=3 and f=5 in Table 1, i.e., f>n. Similarly to the case of faulty 
links, the performance of A t can be analyzed in terms of node failures as follows. 

Lemma 3: Suppose there are g faulty nodes in a Q„, and messages are to be routed from an 
arbitrary node u to another node w, where H(u,w) = k. Let me be the Hamming distance between 

c N T 3-j 

the obstructed node and the destination node w. Then, P(m B =j) < kJ _ 2 if 2<j<k, and 

P(m B = j) = 0 if j = 1 or j > k, where N = 2 n is the total number of nodes in a Q n . 

Proof: Following the same reasoning as in the proof of Lemma 2, we get P(m B >k) = 0. 
Besides, P(m B =l) = 0 since the destination node is assumed to be non-faully. Next, let a 
configuration and the obstructed node be defined analogously to the case of faulty links. (Link 
failures are replaced by node failures.) There are C g N_2 different configurations for a Q n with g 
faulty nodes since the source and destination nodes are assumed to be non-faulty. In order to 
determine P(m B =j), we need to count the number of configurations which lead to the case of 
m B = j. By the same reasoning as the one used in the proof of Lemma 2, the number of such 
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configurations is less than C^lj 3 J . Notice that we use c£Lj 3 * instead of C^ 2 J because the 
obstructed node must be non-faulty, and this lemma thus follows. Q.E.D. 


From Lemma 3 and the reasoning in the proof of Theorem 3, we can obtain Theorem 4 and 
its corollary, showing that A] can also route a message between any pair of non-faulty nodes in 
an injured hypercube via an optimal path with a high probability. 

Theorem 4: Suppose there are g faulty nodes in a Q n . Algorithm A] will route a message 
from a node u to another node w via an optimal path between u and w with a probability greater 

k c N J 3_j 

than 1 - S '~'n- 2 ~ , where H(u,w) = k - 
j=2 C g 

Corollary 4.1: Suppose there are n-1 faulty nodes in a Q n . Algorithm A] will route a mes- 
sage from a node u to another node w via an optimal path between u and w with a probability 

n-2 


greater than 1 - 


(n - l)r 2 (l - r 2 ~ ] ) 
( 2 " - 2)(1 - r2 ) 


, where H(u,w) = k, and r 2 = 


2 n — 3 


Proof: Notice that 


C n -i-j J (n-l)(n-2) • • • (n-j+l)(n-j)(N-n-l) 


-N-2 

"n-1 


(N-2)(N-3) • • • (N-j-2) 

< ( n ~ 1 )( n ~ 2 2_ ' ‘ (n-J+l)(n-j) N-j-2.) 

(N-2)(N-3) • • • (N-j-1) J 

n-1 .m 


N-2 


rf- 


k k n _ i 

By letting g = n-1 in Theorem 4 we get 1 - ^ > 1 - — — rj -1 

j=2 Qili j=2 N - 2 


n - 1 k i ( n_ l)f2(l — r 2 *) 

1 - -5—! L V rC 1 = 1 - where N = 2 n . Q.E.D. 

N-2£ 2 (N-2X1-T2) 


Again, from Theorem 4, we derive Table 2, showing the lower bound of the probability for 
A, to route messages via optimal paths between two nodes of Hamming distance n-1 apart in a 
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q„ with g faulty nodes. Furthermore, as it will be shown below, the expected length of a path 
resulting from the use of A! is very close to that of an optimal path, i.e., the Hamming distance 
between the source and destination nodes. Before analyzing the quality of the paths selected by 
A i, it is necessary to introduce the following proposition. 

Proposition 1: Let { Pi }£, and (q^ be, respectively, two decreasing sequences with 

n-1 n-1^ 

P = On = Suppose Pi < for 1 < i < n-1, then £i( Pi - Pi+i) - SKOi ~ Qi+i)- 
n i=l i=l 

Proof: Let di = qi - pi for 1 < i < n. Then, we get 

ZKOi-Qi+i)- E^Pi-Pi+i) 

i=l i=> 

(Since d ; >0 for 1 <i<n-l.) Q.E.D. 


We can now derive the following important theorem. 

Theorem 5: Let u and v be a pair of nodes with H(u,w) = n in an injured Q n which contains 
n-1 faulty links. Let H, be the length of a path between u and w that is chosen by A,. Then, 

E(AH!> < where E(x) denotes the expected value of a random variable x, and AH! = Hi - n. 


Proof: Notice that P(AH! > 2i) < £P(m A = j). Then, 

j=i 


n-1 


ECAH^E 2 * P(AHj = 2i) 


i=l 
n-1 r 

i=i 


n-1 

<I2i 

i=l 


P(AHi > 2i) - P(AH! > 2(i+l)) j 


IP(m A =j)- n "£ , P(mA=j) 

j=i j=' 


(By Proposition 1.) 


n-1 

= £2i P(m A =n-i) 
i=t 
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n-l ry 

< £2(n-i) — - — (By Lemma 2 and L = n2" .) 

i=i c n L _, 

0- 1 n — 1 

<£2(n-i)r{ 

1- i L 

n-l 

= 2n£r|- 

i=i 

= 2nr! + 2nri(rj + r 2 + • • • + r" -2 ) 


< 2nr! = 


n-l 

2 n_2 


(Since nrj < 1.) Q.E.D. 


Using the same reasoning in the proof of Theorem 5, we have the following corollary for an 
injured Q„ with n-l faulty components. 

Corollary 5.1: Suppose messages are to be routed from an arbitrary node u to another node 
w in an injured Q n which contains n-l faulty nodes. Let H 2 be the length of a path between u and 

w that is chosen by Ai, and let AH 2 = H 2 - n, where H(u,w) = n. Then, E(AH 2 ) < ^ n ( n IKH tl 

(2 n -2)(2 n -3) 

3.3. Depth-First Search Routing and Other Remarks 

Note that in the presence of more than n-l faulty components in a Q n , Algorithm A,, due to 
its simplicity, cannot ensure a faulty component not to be encountered more than once, and thus, 
cannot always lead to a successful message routing. Depth-first search can be applied to deal 
with the problem of routing messages between connected pairs of non-faulty nodes in a hyper- 
cube with an arbitrary number of faulty components [27], In such a case, a set of faulty com- 
ponents encountered before has to be added to the message, and a more complicated procedure is 
required to guide the backtracking whenever it is forced to backtrack from a deadend. Instead of 
keeping track of the entire path traveled, the depth-first search routing can also be implemented 
by using a stack, in which case the operations required for backtracking are simplified, but addi- 
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tional provisions are needed to ensure that a node will not be visited more than once. Note that 
our results on the performance analysis of A! can be extended and applied to the analysis of the 
depth-first search routing. In order not to distract the readers from the main theme of this paper, 
we have not included here such an extended analysis. Interested readers are referred to [28], 

Despite its limiting assumption on the number of faults, using the concept of spare dimen- 
sions, Algorithm A! only needs to keep an n-bit vector (tag), and is shown to be very simple and 
powerful. One cannot overemphasize two important aspects of this algorithm, the ability to route 
messages via optimal paths with a high probability, and the fact that the expected length of a 
resulting path is very close to the Hamming distance between the source and destination nodes. 

However, due to the absence of information at each node on components other than its own 
links, the presence of a certain faulty component is not known until a message gets to the faulty 
component. This may force an intermediate node to use a spare dimension for routing messages 
around the faulty component, thus increasing the length of the actual path taken. Consequently, 
in order to route messages more efficiently, each node needs to be equipped with more informa- 
tion than that on its own communication links such that the faulty components on a path to the 
destination can be bypassed. 

4. ROUTING WITH LIMITED GLOBAL INFORMATION 

As mentioned before, routing efficiency can be improved by increasing each node s infor- 
mation on component failures. We shall examine the effects of failure information at each node 
on the efficiency of routing. First, we shall propose and analyze in Section 4.1 a routing scheme 
in which the condition of each component is not only known to its adjacent nodes but also avail- 
able to those nodes one hop away from that component. Although this scheme improves routing 
efficiency, the paths chosen are not guaranteed to be the shortest. Thus, we shall in Section 4.2 
investigate the information essential for the shortest path routing. The approach of using network 
delay tables [23] to the fault- tolerant routing in hypercubes with an aibitrary number of faults will 
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be discussed in Section 4.3. 

4.1. Propagation of Failure Information to Neighboring Nodes 

Consider a simple modified scheme of Algorithm Aj. In addition to keeping track of the 
condition of its own links, every node also makes this information available to all its neighboring 
nodes. Thus, every node will know not only the condition of its own links but also that of those 
links which are one hop away from it. To use this information, the conditional statement in A] 
marked with (t) is modified in such a way that every intermediate node checks one more hop in 
the coordinate sequence of each message, and uses a spare dimension to bypass faulty com- 
ponents if necessary. From a reasoning similar to the one in the proof of Theorem 2, it can be 
verified that this modified routing scheme can also successfully route a message to any other node 
if the number of faulty components is less than n. More specifically, the performance of this 
modified routing scheme can be described by the following theorem. 

Theorem 6: Suppose a message is to be routed in an injured Q n with f faulty links Irom 
node u to node w where H(u,w) = k. Let m c be the Hamming distance between the obstructed 
node and the destination when each node is informed about the conditions of its own links as well 

as those links one hop away from the node. Then, P(mc=j)^ / Q if 

i=l 

min{l+r-rl ,k-l),P(mc=k)< £Q k CjtS^f / C f L , and P&nH) = 0 otherwise. 

2 i=o 

Proof: Let x denote the obstructed node. First, consider the case when the obstructed node 
is the source node, i.e., m c = k. Clearly, x has k links within SQ(x,w). There will be no optimal 
path from x to w if and only if each of these k links is either faulty or connected to a neighboring 
node with k-1 faulty links toward w. Suppose x has exactly i non-faulty links toward w. Then, 
there are C* ways to determine which of x’s links are non-faulty. For each case, there arc (k-i) + 
i(k-l) specific faulty links, thus resulting in (££+3? different configurations of faulty links. 
The expression for the case of P(mc = k) thus follows. 
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Since every node is informed about the condition of the links one hop away, a message will 
not be routed to any neighboring node whose every link toward w is faulty. Thus, every link of 
the obstructed node toward w can be faulty only when the obstructed node is the source node. 
This is the very reason that different expressions are needed for the cases of 2 < m c < k-1 and 
m c = k. Note, however, that when m c = j * k and the obstructed node x has i non-faulty links 
toward w, these i links are connected to those nodes with j— 1 faulty links toward w. Therefore, 

f 

from the fact that the total number of faulty links i(j— 1) + j - 1 is less than f, we get j < 1 + 1 yl . 
and thus, this theorem follows. Q.E.D. 

To illustrate the improvement of routing efficiency with the above additional information at 
each node, let h A (j) = C f I :" j /C f L and hc(j) = XCjC f ^]^/C f L . As shown in Lemma 2 and 

i=l 

Theorem 6, P(m A =j) < h A (j) and P(mc=j) < hc(j). It can be verified that h c (j) < h A (j) for j > 2, 
meaning that P(mc=j) has a smaller upper bound than P(m A =j). Note, however, that h c (2) > 
h A (2). This is based on the fact that, under our modified routing scheme, an intermediate node 
located two hops away from the destination may foresee the unreachability from itself to the des- 
tination and thus use spare dimensions to bypass those faulty components which could not be 
seen by the same intermediate node under A^ The upper bound of P(m c = 2) is thus greater than 
that of P(m A = 2). (Note, however, that 1^(2) < h A (2) + h A (l).) Therefore, in light of the reason- 
ing in Theorem 5, routing efficiency is improved with the additional information at each node. 

Note, however, that even the above routing scheme cannot guarantee the shortest path rout- 
ing. When a pair of nodes communicate with each other frequently, it is desirable to have the 
shortest path routing, since extra hops will otherwise be traveled during each transmission. 
Although each node can always find a shortest fault-free path from itself to any other node if it 
contains the information on every faulty component, it is impractical to maintain and update such 
information, especially when the size of the network is very large. Therefore, it is important to 
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determine the information required for the shortest path routing. 

42 . Routing via Shortest Paths 

To reduce the amount of information at each node required for the shortest path routing, the 
unnecessary propagation of information on faulty components should be avoided. Notice that a 
faulty node can be viewed as a node with its all links faulty. Therefore, the network information 
kept at each node can be represented by a set of addresses of faulty links. As will be proved later, 
when the number of faulty components is less than n in a Q n , each node does not have to pro- 
pagate the information on faulty links to its neighboring nodes unless these faulty links block all 
the optimal paths from itself to another node. In other words, only when node u finds that all its 
optimal paths to another node, say x, have been blocked by a set of faulty links F, will node u 
propagate the information on F to its neighboring nodes so as to prevent them from choosing 
node u as a next hop toward node x. 

Note that the coordinate sequence of an optimal path from u to w consists of H(u,w) dif- 
ferent numbers representing those dimensions in which u and w differ, meaning that there arc 
H(u,w)! different optimal paths from u to w. Then, we have the following proposition, describing 
the effect of a link failure on the optimal paths between the two nodes. 

Proposition 2: Let N(u\w, r) be the number of optimal paths from u to w which traverse a 
link r. Then, 

(i) . For any link r e SQ(u,w), N(u\ w, r) = [H(u,w)-H(u,x)-l]!H(u,x)!, where x is the one of 

the link r’s two end nodes that is closer to u. 

(ii) . For any link r 4 SQ(u,w), N(u\ w, r) = 0. 

For example, if u = 0100, w = 1001 and r = 0-01, then x = 0101 and N(u\ w, r) = 2! 1 ! = 2. 
On the other hand, if r= 0-11, then N(u\ w, r) = 0, since 0-11 4 SQ(u,w) = **0*. In light of Pro- 
position 2, we can derive the condition for a set of faulty links to block all the optimal paths 
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between any given pair of nodes as follows. Let r x and r y be the relative addresses of any two 
links with respect to node u. Then, the link r y is called a downstream link of r x , written as r x < r y , 
if r x appears before r y in an optimal path from u to r y . Note that r x — x n x n _i x i < r y 
y n y n _i • • • y, iff ( 1 ) Xi < yi for 1 < i < n, where the symbol is ordered such that 0 < - < 1 , and 
(2) r x and r y are the links placed in different dimensions. This fact results in a straightforward 
procedure for determining if a link is a downstream link of another. Each node can thus store the 
relative addresses of faulty links as a partially ordered set [29]. A set of relative addresses of 
faulty links is called a linear set if for any two links r x and r y in the set, either r x < r y or r y < r x . It 
can be verified that an optimal path from u to w will traverse all links in a set of links B only if B 
is a linear set. Let M(u\w, F) be the number of optimal paths from u to w which traverse every 
link in F. Then, following the concept of exclusion and inclusion [30], we obtain the following 
theorem which can determine the number of optimal paths blocked by a set of faulty links. 

Theorem 7: Given a set of faulty links F, the number of the optimal paths from u to w 
blocked by the links in F is N(u\ w, F) = £(-l) i+1 m it where = E M(u\ and B, 

* i=\ B.e FnSQ(u t w) 

denotes a linear set of i links. 

Clearly, the condition for a set of faulty links to block all the optimal paths between two 
nodes u and w is N(u\ w, F) = H(u,w)l. The operations of Algorithm A 2 can be outlined as fol- 
lows. Each node keeps the information about two types of faulty links in the form of relative 
addresses. The first type, denoted by F 0 , is the set of those faulty links whose status has not yet 
been propagated to neighboring nodes, whereas the second type, denoted by F„ is the set of those 
faulty links whose status has already been propagated to neighboring nodes. 

Since relative addresses of faulty links of a node are kept in that node, the information on F 0 
must be modified in accordance with the addresses of receiving nodes when it is propagated to 
neighboring nodes. A formal description of the algorithm for the determination and modification 
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of link failure information is given below. 

Algorithm A 2 : Collection of failure information for the shortest path rou 
Testing 

/* Each node tests all its communication links.*/ 

if (the k-th link of the node is faulty) then 
begin 

F, := F lU {e k }; 
for i := 1 to n do 

if i * k then send e k ©e' along the i-th dimension; 

Propagation; 

end 

Receiving 

/* For each node receiving the information on the failure of the link r.*/ 
if r F then /*F = F 0 ^|F 1 */ 
begin 

F 0 := F 0 i^i {r}; 

Propagation; 

end 

Propagation 

if N((Ar", F) = H(0",r)! then 
begin 

F 0 :=F 0 -{r); 

Fi :=F lU {r); 

/* Propagate the information on the failure of r to neighboring nodes.*/ 
for i ;= 1 to n do send r©e' along the i-th dimension; 

/* Check if propagation of information on other faulty links is necessary.*/ 
for all r x e F 0 and r e SQ(0", r x ) do 
if N(0\ r x , F) = 11(0", r")! then 
begin 

/* Propagate the information on the failure of r to neighboring nodes.*/ 
for i := 1 to n do send r^fe 1 along the i-th dimension; 

F 0 :=F 0 - {r x }; 

F t :=Fj u {r x }; 

end 

end 
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When a node receives the information on the failure of link r, it will update its F 0 and Fi 
accordingly, and check if any further propagation of information on other link failures m F 0 is 
required. For example, consider the Q 4 in Fig. 4. By executing A* each node can determine 
F = f oU F, as well as the information to be propagated to neighboring nodes, F,. Table 3 shows 
the information to be kept in each node. Notice that the faulty links are represented in each node 
by their relative addresses with respect to the node. Then, we have the theorem below. 

Theorem 8: Under Algorithm A 2 , every node can obtain the failure information essential 
for the shortest path routing as long as the number of faulty components is less than n. 

Proof: Notice that the necessary and sufficient condition for all the optimal paths from node 
u to node w to be blocked is that “for all z e SQ(u,w) reachable from u via an optimal path, then 
there is no optimal path from z to w.” Since every node propagates the corresponding failure 
information to its neighboring nodes if all its optimal paths to a certain node are blocked, the fact 
that every node will know if all its optimal paths to a certain node are blocked can be proved by 

induction. 

When node u finds all its optimal paths to w are blocked, there are at least H(u,w) = k faulty 
components in SQ(u,w). Note that there are still n - k disjoint paths of length k+2 via the neigh- 
boring nodes of u which are not within SQ(u,w), and at least one of them is fault-free because 
there are at most n - 1 faulty components in the Q n . Since those neighboring nodes not having 
any optimal path to w will propagate the corresponding failure information to u to prevent u from 
choosing one of them as a next hop, this theorem follows. Q.E.D. 

Theorems 1 and 8 lead to the following corollary. 

Corollary 8.1: Algorithm A 2 can route a message from node u to node w in H(u,w)+2 hops 
as long as the number of faulty components is less than n. 
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When a node needs to send a message to another node, it will use its information on faulty 
components to determine the coordinate sequence of a shortest fault-free path to the destination 
node as if it had the information on every faulty component in the entire network. Then, accord- 
ing to the first entry of the coordinate sequence, the source node will determine the next hop of 
the message and its associated coordinate sequence. On the other hand, when a node receives a 
message and the coordinate sequence of the remaining part of the path, it will check whether the 
remaining path contains faulty links and permute the order of entries in the coordinate sequence 
to bypass the faulty components, if necessary. 

For example, consider the injured Q 4 in Fig. 4. The source node is not aware of any faulty 
link, and thus, routes a message (3, [2,3,4], fin) to 0111. However, the node 0111 will find the 
path [2,3,4] is faulty, since the path will encounter the faulty link 0-01 whose relative address is 
0-01/oiit =0-10. Thus, a new non-faulty path [3,2,4] is determined by 01 1 1. The message will be 
routed to 0011, and then to the destination 1001 via 1011. The length of the resulting path is 4. 
This is far less than the length of the path determined by A lt 8 . 

It is interesting to see that the information about an isolated faulty link needs to be pro- 
pagated only to its neighboring nodes, whereas the information about clustered faulty links has to 
be propagated a little farther to ensure each message to be routed via a shortest path. For exam- 
ple, node 1 1 1 1 has to be informed by node 1101 about the failure 0-01 (two hops away) and 1 -01 
(one hop away), since all the optimal paths from 1 101 to 0001 are blocked by the failure of 0-01 
and 1-01. This agrees well with our intuition, since clustered faulty components are likely to 
block more optimal paths between a pair of nodes, and thus, have to be kept by those nodes far 
away from them to achieve the shortest path routing. Clearly, when the size of the hypcrcubc 
increases, faulty components will tend to spread out and the size of the zone influenced by a 
faulty component will become rather small relative to the size of the entire network. 
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Notice, however, when the number of faulty components is more than n-1 in a Q n , each 
node may not be able to gather enough information required for the shortest path routing, since 
too many faulty components may block the propagation of failure information. 

4.3. Routing with Delay Tables 

In the presence of more than n-1 faulty components in a Q n , the concept of using network 
delay tables, which was previously used in the ARPANET [23], can be applied to accomplish the 
shortest path routing. Under the algorithms in [23], every node maintains a network delay table 
to record the shortest delay via each link of the node to every other node. When a node is to send 
a message to another node, it will check its network delay table and determine the next hop of the 
message for the shortest path routing. A minimal delay vector in a node, which contains the 
delays of the shortest paths from that node to all the other nodes, is periodically passed to all of 
its adjacent nodes as routing information. After receiving minimal delay vectors from its adja- 
cent nodes, every node will update its network delay table accordingly. For example, the network 
delay tables for nodes 000, 100 and 101 in Fig. 6 are given in Table 4a, 4b and 4c, respectively. 
The routing information generated by node 100 is also shown in Table 4d. As we pointed out in 
[31], looping may occur in the presence of component failures when this routing scheme is used. 
The approach of using high-order routing strategies [32] can be applied to eliminate the looping. 

It is worth mentioning that when the number of faulty components is less than n in a Q n , 
Algorithm A 2 is shown to be capable of routing messages via shortest paths without using net- 
work delay tables, and is thus preferred over the one based on network delay tables. Note that it 
becomes very costly to maintain and update network delay tables as the size of a hypcrcubc gets 
large. It is therefore advantageous to use A 2 , whenever possible. 


25 



February 9, 1989 


5. CONCLUSION 

In this paper, we have proposed and analyzed two distributed routing schemes (A t and A 2 ), 
and introduced two more schemes (using depth-first search and network delay tables), to route 
messages in injured hypercube multicomputers. Aj is very simple and powerful. It requires each 
node to know only the failure of its own links and uses the abundant connections in hypercubcs. 
Performance of this scheme has been rigorously analyzed; we showed that this scheme is not only 
capable of routing messages successfully in an injured Q„ when the number of component 
failures is less than n, but also able to choose a shortest path with a very high probability. To 
handle the case when the total number of faults is greater than n-1 in a Q n , we introduced a rout- 
ing scheme based on depth-first search. However, due to the insufficient amount of information 
on faulty components, these two schemes do not always guarantee the shortest path routing. 

To ensure the shortest path routing, we proposed A 2 which requires each node to be 
equipped with more information than that on its own links. We developed a method which deter- 
mines the failure information essential for each node to guarantee the shortest path routing. It 
turns out that each node is required to know only the condition of a relatively small number of 
components in its vicinity. In case there are more than n-1 faults in a Q n , we can use a more 
expensive routing scheme based on network delay tables. 

Due to their simplicity and/or power, the fault-tolerant routing algorithms derived in this 
paper have high potential use for the growing number of fault-tolerant applications on large 
hypercube multicomputers. 
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1 

2 

3 

4 

5 

6 

3 

0.916 

0.818 

0.700 

0.557 

0.386 

0.182 

4 

0.968 

0.935 

0.900 

0.862 

0.821 

0.777 

5 

0.987 

0.974 

0.961 

0.948 

0.934 

0.920 


Table 1 Lower bounds of the probabilities obtained from Theorem 3 for the optimal path rout- 
ing between two nodes of distance n apart in the presence of link failures. 


n 

2 

3 

4 

5 

6 

4 

0.989 

0.967 

0.934 

0.891 

0.838 

5 

0.998 


0.986 

0.977 

0.965 

5 

0.999 

0.998 

0.996 

0.994 

0.992 


Table 2. Lower bounds of the probabilities obtained from Theorem 4 for the optimal path rout- 
ing between two nodes of distance n apart in the presence of node failures. 

























Table 3. Information in each node generated by Algorithm A 2 
for the injured hypercube in Fig. 4. 





destination 

dimension 

001 * 

010 

011 

100 

101 

110* 

111 

1 (001) 

oo 

OO 

OO 

OO 

oo 

OO 

OO 

2 (010) 

3 

1 

2 

3 

4 

2 

3 

3 (100) 

5 

3 

4 

1 

2 

4 

3 


(a). Network delay table of node 000. 


^destination 
dimension — , 

000 

001 * 

010 

Oil 

101 

110 * 

111 

1 (101) 

3 

4 

4 

3 

1 

3 

2 

2 (110) 

OO 

OO 

OO 

OO 

OO 

OO 

OO 

3 (000) 

1 

4 

2 

3 

3 

3 

4 


(b). Network delay table of node 100. 


dimension" 

000 

001 * 

010 

Oil 

100 

110* 

111 

1 (100) 

2 

5 

3 

4 

1 

4 

3 

2 (111) 

4 

3 

3 

2 n 

3 

2 

1 

3 (110) 

OO 

OO 

OO 

OO 

OO 

OO 

OO 


(c). Network delay table of node 101. 


000 

001 * 

010 

Oil 

101 

110 

111 

1 

4 

2 ^ 

3 

1 

3 

2 


(d). Routing information generated by node 100. 


Table 4. Network delay tables for hypercube nodes in Fig. 6. 
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Figure 4 . An injured Q 4 where links 0-01, 
1-01 and 100- are faulty. 
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Figure 5 . An injured Q 4 where links 0-11, 
-Oil and 111- are faulty. 
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Figure 6 . An example Q 3 for the routing scheme 
based on the minimal delay tables. 



